.1 



United States R\tent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 

United States Potent and Trademark Office 

Address: COMMISSIONER OF PATENTS AND TRADEMARKS 

Washington, D.C. 20231 

www.uspto.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. 



CONFIRMATION NO. 



09/666,281 



09/21/2000 



Jin Soo Lee 



III-019 



8469 



7590 



01/31/2003 



Fleshner & Kim LLP 
POBox 221200 
Chantilly,VA 20153-1200 



EXAMINER 



MAHMOUDI, HASSAN 



ART UNIT 



PAPER NUMBER 



2175 

DATE MAILED: 01/31/2003 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 07-01) 



Office Action Summary 



Application No. ^ 

09/666,281 


pVpplicant(s) 

LEE ET AL. 


Examin r 

Tony Mahmoudi 


Art Unit 

2175 





Th MAILING DATE of this communication appears on th cov rsh et with the correspondenc addr ss 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)S Responsive to communication(s) filed on 15 November 2002 . 
2a)Q This action is FINAL. 2b)S This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) E3 Claim(s) 1-20 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

Q)M Claim(s) 1-4. 6. and 14-20 is/are rejected. 

7) ^ Claim(s) 5. and 7-13 is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)Q objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

1 1) D The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13/^Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
aJ&AII b)D Some*c)D None of: 

1 .fe* Certified copies of the priority documents have been received. 

2.Q Certified copies of the priority documents have been received in Application No. . 



3.D Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. (^^[^&))^ZJ^ 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 . D0V POTOVlCI 
Attachment^ SUPERVISORY PATENT EXAMINER 

1 ) [X] Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-4^^at)Qf-f}^) C _^f^ R 21 00 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) □ Notice of Informal Patent Application (PTO-152) 

3) □ Information Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) □ Other: 



U.S. Patent and Trademark Office 

PTO-326 (Rev. 04-01) 



Office Action Summary 



Part of Paper No. 5 



Application/Control Number: 09/666,281 
Art Unit: 2175 



Page 2 



DETAILED ACTION 

Remarks 

I 

1. In response to communications filed on 15-November-2002, claims 1-3 and 14-15 are 
amended, and new claims 18-20 are added per applicant's request. Therefore, claims 1-20 
are pending in the application. 

Claim Rejections - 35 USC § 103 

2. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 
rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

3. Claims 1-3 and 6 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ma et al 
(U.S. patent No. 6,347,313) in view of Liddy et al (U.S. patent No. 6,304,864.) 

As to claim 1, Ma et al teaches a multimedia data structure reflecting change of a user 
relevance feedback (see column 1, lines 5-10) for determining weights of image features used for 
an image search, comprising: 

(a) information describing at least one feature of a certain image (see column 3, lines 18-23); 

and 
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(b) recent user feedback information (see column 3, lines 37-56) based on user relevance 
feedback (see column 6, lines 45-50.) 

Ma et al does not teach: 

(c) whole feedback information based on the user relevance feedback. 

Liddy et al teaches a system for retrieving multimedia information (see Abstract), in which 
she teaches whole feedback information based on the user relevance feedback (see column 12, 
lines 9-24, and see column 13, lines 26-38, where "whole feedback" is read on "relevance 
feedback" on a "periodic time interval".) 

Therefore, it would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Ma et al to include whole feedback information based 
on the user relevance feedback. 

It would have been obvious to a person having ordinary skill in the art at the time the 
invention was made to have modified Ma et al by the teaching of Liddy et al because having 
whole feedback information based on the user relevance feedback, would enable the system to 
define periodic intervals automatically, or as set by the user, to allow accumulation of all 
relevance feedback on a particular object (image) to be captured, in order to categorize the 
objects (image) based on whole user relevance feedback. 

As to claim 2, Ma et al as modified teaches wherein the recent user feedback information is 
determined for a predetermined time period (see Liddy et al, column 12, lines 13-14, where 
"predetermined time period" is read on "interval may be 15 minutes") or by a predetermined 



Application/Control Number: 09/666,28 1 Page 4 

Art Unit: 2175 

frequency (see Liddy et al column 12, lines 21-22, where "predetermined frequency" is read on 
"X number of documents".) 

As to claim 3, Ma et al as modified teaches wherein the recent user feedback information is a 
weight value learned by the user relevance feedback or a similar image information (see Ma et 
al, column 6, lines 49-50, and see column 8, lines 4-10), and the whole feedback information is 
represented by a weight value learned by previous feedback (see Liddy et ah column 13, lines 
30-33.) 

As to claim 6, Ma et al as modified teaches the method comprising: 

representing the recent user feedback information by a similar image list (see Liddy et al 

column 8, lines 60-64); and 

reflecting a recent user feedback pattern by the similar image list (see Liddy et al , column 3, 

line 64 trough column 4, line 9), using a queue algorithm (see Liddy et ah column 10, lines 34- 

38.) 

4. Claims 4 and 14-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ma et al 
(U.S. patent No. 6,347,3 13) in view of Liddy et al (U.S. patent No. 6,304,864), as applied to 
claims 1-3 and 6 above, and further in view of Cohen (U.S. Patent No. 6,067,539.) 

As to claim 4, Ma et al as modified does not teach the data structure further comprising 
recent user feedback reliability information representing how reliable the recent user 
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feedback information is, and whole feedback reliability information representing how 
reliable the whole feedback information is. 

Cohen teaches an intelligent information retrieval system (see Abstract), in which he 
teaches: recent user feedback reliability information representing how reliable the recent 
user feedback information is (see column 7, lines 51-56), and whole feedback reliability 
information representing how reliable the whole feedback information is (see column 2, lines 
45-64, where "whole feedback" is read on "updating the score with scores received on 
previous message".) 

Therefore, it would have been obvious to a person having ordinary skill in the art at the 
time the invention was made to have modified Ma et al as modified to include recent user 
feedback reliability information representing how reliable the recent user feedback 
information is, and whole feedback reliability information representing how reliable the 
whole feedback information is. 

It would have been obvious to a person having ordinary skill in the art at the time the 
invention was made to have modified Ma et al as modified, by the teaching of Cohen , 
because recent user feedback reliability information representing how reliable the recent user 
feedback information is, and whole feedback reliability information representing how 
reliable the whole feedback information is, would increase the efficiency and accuracy of the 
entered feedback and allow data (images) with the most relevant/reliable user feedback to 
receive a higher rank/weight for presentation to the user than data (images) with a lower 
reliability score. 
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As to claim 14, Ma et al teaches a method of determining weights of image features used for 
an image search based on user relevance feedback (see Abstract), comprising: 

(a) providing a multimedia data structure (see column 1, lines 5-10) including information 
describing the features of a certain image (see column 3, lines 18-23), and recent user feedback 
information for the image (see column 3, lines 37-56, and see column 6, lines 45-50); 

(b) updating the recent user feedback information (see column 2, lines 28-33); 

(c) determining weights of the image features in proportion to the reliabilities of the recent 
feedback information (see column 6, lines 11-18.) 

Ma et al does not teach: whole feedback information for the image; and determining weights 
of whole feedback information, or both the recent feedback information and the whole feedback 
information. 

Liddy et al teaches a system for retrieving multimedia information (see Abstract), in which 
she teaches: whole feedback information for the image (see column 12, lines 9-24, and see 
column 13, lines 26-38, where "whole feedback" is read on "relevance feedback" on a "periodic 
time interval"); and determining weights of whole feedback information, or both the recent 
feedback information and the whole feedback information (see column 13, lines 15-18.) 

Therefore, it would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Ma et al to include whole feedback information for the 
image; and determining weights of whole feedback information, or both the recent feedback 
information and the whole feedback information. 

It would have been obvious to a person having ordinary skill in the art at the time the 
invention was made to have modified Ma et al by the teachings of Liddv et al because obtaining 
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whole feedback information for the image; and determining weights of whole feedback 
information, or both the recent feedback information and the whole feedback information, would 
enable the system to define periodic intervals automatically, or as set by the user, to allow 
accumulation of all relevance feedback on a particular object (image) to be captured, in order to 
categorize the objects (image) based on whole user relevance feedback and based on the weights 
(ranking) of relevance of the entered feedback. 

Ma et al as modified still does not teach reliability information corresponding to the 
recent user feedback information and whole feedback information; and updating whole 
feedback information and their reliabilities by learning them in response to the user relevance 
feedback. 

Cohen teaches an intelligent information retrieval system (see Abstract), in which he 
teaches reliability information corresponding to the recent user feedback information (see 
column 7, lines 51-56), and whole feedback information (see column 2, lines 45-64, where 
"whole feedback" is read on "updating the score with scores received on previous message"); 
and updating whole feedback information and their reliabilities by learning them in response n 
to the user relevance feedback (see column 7, lines 36-38.) 

Therefore, it would have been obvious to a person having ordinary skill in the art at the 
time the invention was made to have modified Ma et al as modified to include reliability 
information corresponding to the recent user feedback information and whole feedback 
information; and updating whole feedback information and their reliabilities by learning 
them in response to the user relevance feedback. 
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It would have been obvious to a person having ordinary skill in the art at the time the 
invention was made to have modified Ma et al as modified, by the teaching of Cohen , 
because reliability information corresponding to the recent user feedback information and 
whole feedback information; and updating whole feedback information and their reliabilities 
by learning them in response to the user relevance feedback, would increase the efficiency 
and accuracy of the entered feedback and allow data (images) with the most relevant/reliable 
user feedback to receive a higher rank/weight for presentation to the user than data (images) 
with a lower reliability score. 

As to claim 15, Ma et al as modified teaches wherein the recent user feedback 
information is represented by a weight value learned by the user relevance feedback or a 
similar image information (see Ma et al column 6, lines 49-50, and see column 8, lines 4- 
10), and the whole feedback information is represented by a weight value learned by 
previous feedback (see Liddy et al column 13, lines 30-33.) 

As to claim 16, Ma et al as modified teaches wherein the reliability of the recent user 
feedback information (see Cohen , column 7, lines 51-56) is determined in proportion to a 
consistency of a recently used pattern or feedback (see Liddy et al column 3, line 64 trough 
column 4, line 9.) 
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As to claim 17, Ma et al as modified teaches wherein the reliability of the whole feedback 
information is determined in proportion to the number of feedback concerned in learning (see 
Cohen , column 9, lines 32-51.) 

As to claim 18, Ma et al as modified teaches wherein the reliability of the whole feedback 
information is responsive to recorded usage records wherein the recorded user usage records 
provide feedback to the reliability of the whole feedback information without user interaction 
(see Cohen , column 4, lines 31-39.) 

As to claim 19, the applicant is directed to the remarks and discussions made in claim 14 
above. 

As to claim 20, Ma et al as modified teaches wherein the reliability information indicates 
reliability of both the user feedback information (see Cohen , column 7, lines 51-56) and the 
whole feedback information (see Cohen , column 2, lines 45-64, where "whole feedback" is 
read on "updating the score with scores received on previous message.) 

Allowable Subject Matter 
5. Claims 5 and 7-13 are objected to as being dependent upon a rejected base claim, but would 
be allowable if rewritten in independent form including all of the limitations of the base 
claim and any intervening claims. 
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6. The following is a statement of reasons for the indication of allowable subject matter: 

The prior art of record, Ma et al (U.S. Patent No. 6,347,3 13), Liddv et al (U.S. Patent No. 
6,304,864), and Cohen (U.S. Patent No. 6,067,539) do not disclose, teach, or suggest the claimed 
limitations of (in combination with all other features in the claims): 

wherein the recent user feedback reliability information is expressed by 



where, N is the number of feedback, m is the number of images in the similar image list, and 
n(i) is the number of feedback given to the i-th image, as claimed in claim 5. 

The prior art of record, Ma et al (U.S. Patent No. 6,347,3 13), Liddv et al (U.S. Patent No. 
6,304,864), and Cohen (U.S. Patent No. 6,067,539) do not disclose, teach, or suggest the claimed 
limitations of (in combination with all other features in the claims): 

(b)if it is checked that the corresponding object does not exist in the current queue, inputting 
the corresponding object to an uppermost space of a queue entrance, setting the number of 
feedback of the corresponding object to "1", and shifting objects existing in the queue to lower 
positions by one space; 





i - 
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(c) if it is checked that the corresponding object exists in the current queue, increasing the 
number of feedback of the corresponding object, and shifting the objects existing in the queue to 
upper positions by "N"; and 

(d) if any object is shifted to the lower position over a size of the queue at the respective 
steps, deleting the corresponding object from the queue, as claimed in claim 7. 

Claims 8-10 are objected to as being dependent upon the objected to dependent claim 7. 

The prior art of record, Ma et al (U.S. Patent No. 6,347,3 13), Liddv et al (U.S. Patent No. 
6,304,864), and Cohen (U.S. Patent No. 6,067,539) do not disclose, teach, or suggest the claimed 
limitations of (in combination with all other features in the claims): 

wherein the recent user feedback information is represented as a similar image list, and the 
similar image list has an image list structure composed of a similar image identification, a score 
reflecting the current feedback, and a waiting duration representing a time period between the 
final feedback time point and the present time point, as claimed in claim 1 1 . 

Claims 12-13 are objected to as being dependent upon the objected to dependent claim 11. 

Response to Arguments 
7. Applicant's arguments filed on 15-November-2002 with respect to claims 1-17 have been 
fully considered but they are moot in view of the new grounds for rejection. 
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Conclusion 



8. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

The following patents are cited to further show the state of art with respect to methods 
and systems of searching images in an image database, providing feedback on the image 
characteristics, and measuring feedback reliability in general: 

U.S. Patent No. 6,173,275 to Caid et al . 

9. Any inquiries concerning this communication or earlier communications from the examiner 
should be directed to Tony Mahmoudi whose telephone number is (703) 305-4887. The 
examiner can normally be reached on Mondays-Fridays from 08:00 am to 04:30 pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Dov Popovici, can be reached at (703) 305-3830. 
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